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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 
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Panel  13.  Risk  and  Reward  in  Defense  Contracting 


Wednesday,  May  16,  2012 

3:30  p.m.  - 
5:00  p.m. 

Chair:  The  Honorable  Brian  Miller,  Inspector  General,  U.S.  General  Services 
Administration 

The  Excessive  Profits  of  Defense  Contractors:  Evidence  and  Determinants 

Chong  Wang  and  Joseph  San  Miguel 

Naval  Postgraduate  School 

Fixed-Price  Development  Contracts:  A  Historical  Perspective 

William  Lucyshyn,  Jacques  S.  Gansler,  and  Jiahaun  Lu 

University  of  Maryland 

A  Quantitative  Risk  Analysis  of  Deficient  Contractor  Business  System 

William  Fast,  Naval  Postgraduate  School 

Brian  Miller — The  U.S.  Senate  confirmed  Brian  D.  Miller  as  the  Inspector  General  of  the  U.S. 

General  Services  Administration  on  July  22,  2005. 

As  Inspector  General,  Miller  leads  more  than  300  auditors,  special  agents,  lawyers,  and  support 
staff  in  conducting  nationwide  audits  and  investigations.  As  a  national  leader  in  the  fight  against 
procurement  fraud,  Miller  participates  in  the  U.S.  Attorney  General’s  Financial  Fraud  Enforcement 
Task  Force  and  partners  with  federal,  state,  and  local  officials  to  share  information  to  detect, 
investigate,  and  prevent  procurement,  Recovery  Act,  and  grant  fraud.  Miller  is  a  frequent  speaker  at 
conferences,  task  force  meetings,  and  regional  working  groups,  and  he  testifies  regularly  before 
Congress. 

Before  becoming  Inspector  General,  Miller  worked  for  the  U.S.  Department  of  Justice  for  15 
years,  beginning  in  the  Office  of  Policy  Development.  Attorney  General  Janet  Reno  appointed  him  as 
an  assistant  U.S.  attorney  for  the  Eastern  District  of  Virginia,  where  he  concentrated  on  procurement, 
grant,  and  health  care  fraud  cases.  In  2001 ,  he  served  as  the  senior  counsel  to  the  deputy  attorney 
general  and  special  counsel  for  health  care  fraud  for  the  U.S.  Department  of  Justice.  In  2002,  he 
returned  to  the  U.S.  Attorney’s  Office  to  serve  as  counsel  to  the  United  States  attorney,  while 
continuing  grand  jury,  trial,  and  appellate  responsibilities  as  an  assistant  U.S.  attorney. 

Miller  strives  to  provide  aggressive,  strategic,  and  creative  leadership  by  developing  new  ways  to 
fight  fraud.  In  2010,  he  established  the  Government  Infrastructure  Protection  Initiative  within  his  Office 
of  Investigations  to  protect  the  federal  procurement  process  and  supply  chain  from  the  dangers 
associated  with  counterfeit  products.  In  this  effort,  he  partnered  with  the  National  Intellectual  Property 
Rights  Coordination  Center.  In  2008,  he  created  a  forensic  auditing  unit  and  sponsored  federal 
forums  to  promote  forensic  auditing  as  a  tool  to  analyze  databases  and  to  detect  fraud.  In  2006,  Miller 
was  named  vice  chair  of  the  National  Procurement  Fraud  Task  Force,  a  task  force  of  the  Department 
of  Justice,  law  enforcement  offices,  and  several  inspectors  general.  As  co-chair  of  the  Legislation 
Committee,  he  played  a  key  role  in  amending  the  Federal  Acquisition  Regulation  to  require 
contractors  to  report  overpayments  and  crimes.  He  was  also  a  principal  author  of  a  legislative  and 
regulatory  reform  white  paper,  which  resulted  in  legislation,  regulation,  and  a  national  debate  on 
issues  related  to  procurement  fraud. 

Miller  has  received  notable  recognition  for  his  service  as  Inspector  General.  He  was  recognized 
by  Ethisphere  Magazine  as  the  12th  “most  influential  person  in  business  ethics”  by  a  worldwide  panel 
of  experts.  He  was  named  among  “Those  Who  Dared:  30  Officials  Who  Stood  Up  for  Our  Country,”  a 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  381  - 


special  report  of  Citizens  for  Responsibility  and  Ethics  in  Washington,  a  national  advocacy 
organization.  Miller  also  received  the  Attorney  General’s  Award  for  Distinguished  Service. 

Miller  has  a  Juris  Doctor  from  the  University  of  Texas. 
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Quantitative  Risk  Analysis  of  Deficient  Contractor 

Business  Systems 


William  Fast — COL  Fast,  USA  (Ret.),  facilitates  acquisition  and  program  management  courses  at  the 
Naval  Postgraduate  School.  He  also  writes  and  speaks  on  various  management  topics  and  provides 
consultation  services  to  defense  acquisition  programs.  From  2006-2010  COL  Fast  taught  program 
and  financial  management  courses  at  the  Defense  Acquisition  University,  [wrfast@nps.edu] 

Abstract 

This  research  reviews  quantitative  risk  models  to  determine  how  to  best  portray  the  risk  to  the 
government  of  deficient  contractor  business  systems.  One  model,  operational  value  at  risk 
(VaR)  is  proposed  as  the  model  for  use  by  a  government  contracting  officer  when  justifying 
the  withholding  of  contractor  payments  for  a  “significant  deficiency”  in  the  contractor's  earned 
value  management  (EVM)  system  (per  DFARS  252.234.7005,  effective  August  1 6,  201 1 ). 

Research  Issue 

The  director  of  the  Earned  Value  Management  Division  of  the  Defense  Contract 
Management  Agency  (DCMA)  asked  for  assistance  in  developing  a  method  that 
administrative  contracting  officers  (ACO)  can  use  to  assess  risk  associated  with  non- 
compliant  contractor  business  systems.  A  recent  rule  change  in  the  Defense  Supplement  to 
the  Federal  Acquisition  Regulation  (DFARS,  2011)  permits  an  ACO  to  withhold  up  to  10%  of 
contract  payments  for  a  “significant  deficiency”  in  a  contractor’s  business  system  that 
creates  risk  to  the  government.  The  research  issue  is  how  to  objectively  and  quantitatively 
portray  that  risk  in  a  way  that  supports  a  monetary  withhold  decision  and  can  withstand 
push-back  (to  include  litigation)  from  the  defense  contractor. 

Background 

Patterned  after  the  Truman  Committee  of  the  World  War  II  era,  the  Commission  on 
Wartime  Contracting  in  Iraq  and  Afghanistan  cited  numerous  deficiencies  in  business 
systems  used  by  companies  executing  contracts  in  support  of  those  wars  (Commission  on 
Wartime  Contracting,  2009).  As  a  result  of  the  commission’s  hearing  before  the  Congress  in 
August  2009,  Senate  Bill  3454  was  introduced  and  ultimately  found  its  way  into  the  Ike 
Skelton  National  Defense  Authorization  Act  (NDAA)  for  FY  2011,  directing  that 

the  Secretary  of  Defense  develop  and  initiate  a  program  for  the  improvement 
of  contractor  business  systems  to  ensure  that  such  systems  provide  timely, 
reliable  information  for  the  management  of  Department  of  Defense  programs 
by  the  contractor  and  by  the  Department.  (NDAA  for  FY201 1 ,  §  893) 

The  legislation  was  precise  in  defining  a  “contractor  business  system”  and  a 
“covered  contract”  but  only  loosely  defined  a  “significant  deficiency”  (see  Table  1). 
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Table  1.  Definition  of  Terms,  Contractor  Business  Systems 

(NDAA  for  FY2011,  §  893) 


Term 

Definition 

Contractor 

Business 

System 

•  accounting  system 

•  estimating  system 

•  purchasing  System 

•  earned  value  management  system 

•  material  management  and  accounting  system 

•  property  management  system 

Covered 

Contractor 

A  contractor  that  is  subject  to  the  cost  accounting  standards  under  section  26 
of  the  Office  of  Federal  Procurement  Policy  Act  (41  U.S.C.  422) 

Covered 

Contract1 

•  cost-reimbursement  contract 

•  incentive-type  contract 

•  time-and-materials  contract 

•  labor-hour  contract 

Significant 

Deficiency 

“...shortcoming  in  the  system  that  materially  affects  the  ability  of  officials  of  the 
Department  of  Defense  and  the  contractor  to  rely  upon  information  produced 
by  the  system  that  is  needed  for  management  purposes.” 

Acting  on  the  2009  Interim  Report  of  the  Commission  on  Wartime  Contracting,  the 
Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology, 
and  Logistics),  published  a  draft  rule  on  contractor  business  systems  in  January  2010.  This 
first  attempt  at  a  contracting  rule  tried  to  explain  what  constitutes  an  acceptable  business 
system  and  the  percent  of  payments  that  would  be  withheld  for  deficiencies  in  single  or 
multiple  business  systems.  In  response,  the  Department  received  370  comments  from  25 
respondents  (Brodsky,  2010). 

Another  draft  of  the  proposed  rule  was  issued  for  comment  in  December  2010.  This 
draft  clarified  how  Department  officials  would  determine  when  contractor  business  systems 
are  deficient,  and  it  lowered  from  20%  to  10%  the  amount  of  payments  that  the  Department 
could  withhold  from  contractors  when  deficiencies  go  uncorrected  (Brodsky,  2010). 
Respondent  comments  to  the  December  2010  draft  rule  for  contractor  business  systems 
ranged  over  34  different  topic  areas.  Some  of  the  more  significant  objections  to  this  draft 
rule  were  the  following: 

•  The  rule  fails  to  offer  any  guidance  to  the  contracting  officer  for  describing  a 
“significant  deficiency.” 

•  There  is  the  potential  for  inconsistent  application  of  business  system  criteria 
when  determining  a  “significant  deficiency.” 

•  The  Defense  Contract  Management  Agency  (DCMA)  and  the  Defense  Contract 
Audit  Agency  (DCAA)  are  under-resourced  to  implement  the  rule. 

By  May  18,  2011,  an  interim  rule,  with  comments  from  interested  respondents,  was 
published  in  the  Federal  Register  (DFARS,  2011,  Case  2009-D038,  pp.  28856-28879).  The 
interim  rule  took  effect  on  contracts  awarded  on  or  after  August  16,  201 1 . 


1  “The  [interim]  rule  [dated  May  18,  2011]  has  been  tailored  to  comply  with  section  893  of  the  FY11  NDAA.  DoD 
has  interpreted  the  definition  of  ‘covered  contract’  to  include  CAS-covered  cost  type  contracts  as  well  as  CAS- 
covered  fixed-price  contracts  and  performance-based  contracts  since  section  893  also  allows  up  to  10  percent  of 
progress  payments  and  performance-based  payments  to  be  withheld”  (Federal  Register,  201 1 ,  p.  28862). 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-402- 


In  addition,  the  DCMA  has  developed  an  implementation  procedure  to  guide  its 
administrative  contracting  officers  who  will  be  enforcing  the  rule  (Contractor  Business 
Systems,  Instruction  Folder  Number  301  to  the  DCMA  [2012a]  Guidebook). 

To  date,  only  one  contractor  has  had  payments  withheld  under  the  new  rule.  Starting 
in  March  2012,  2%  of  payments  to  Lockheed  Martin  Corporation’s  aeronautics  division  were 
withheld  due  to  deficiencies  in  its  earned  value  management  system  (EVM  system).  The 
withholds  total  about  $1  million  per  month  from  billings  on  Lockheed’s  latest  production 
contract  for  30  F-35  fighters.  The  total  contract  value  is  about  $4  billion  (Cappacio,  2012).  If 
the  EVM  system  deficiencies  remain  uncorrected  through  the  entire  period  of  performance 
of  the  contract,  the  total  withhold  could  be  about  $80  million. 

The  DCMA  decertified  Lockheed’s  aeronautics  division  EVM  system  in  October 
2010,  about  10  months  before  the  business  systems  rule  went  into  effect.  At  that  time,  the 
DCMA  had  determined  that  19  of  32  EVM  system  guidelines  were  not  being  followed  (see 
Appendix  A  for  the  32  EVM  system  guidelines).  Owing  to  the  fact  that  Lockheed  has  made 
corrections  and  improvements  to  its  EVM  system  since  that  time,  the  Pentagon  payment 
withholdings  were  set  at  2%  rather  than  the  5%  maximum  permitted  by  law  for  a  significant 
deficiency  in  a  single  business  system  (Cappacio,  2012). 

Literature  Review 

Galway  (2004)  presents  a  critical  review  of  the  use  of  quantitative  risk  analysis  in 
system  development  projects  like  those  managed  by  the  Department  of  Defense.  He 
concludes  that  quantitative  risk  analysis  can  be  a  valuable  tool;  however,  it  is  not  well 
understood  by  project  managers,  not  well  integrated  into  project  management,  and  hard  to 
explain  to  senior  decision-makers.  He  found  few  empirical  examples  attesting  to  the 
usefulness  of  quantitative  risk  analysis  in  complex  development  projects  (Galway,  2004,  p. 
19). 

Qualitative  Risk  Analysis  Prevails  Within  DoD 

Galway’s  observations  track  with  current  Department  of  Defense  acquisition  policy 
and  guidance.  Quantitative  assessment  of  risk  consequences  are  mentioned  only  once  in 
the  Risk  Management  Guide  for  DoD  Acquisition  (USD[AT&Lj,  2006,  p.  33).  In  the  current 
version  of  the  Defense  Acquisition  Guidebook ,  the  phrase  “quantitative  risk  analysis”  is  used 
only  three  times  (all  uses  pertain  to  cost  estimating),  and  there  are  no  practical  examples 
provided  to  explain  the  analysis  method  to  the  reader  (DoD,  2012,  pp.  138,  158). 

However,  Galway  (2004)  does  bring  insight  to  the  challenge  of  quantifying  business 
system  risks  by  suggesting  that  all  risks  should  be  mapped  to  cost,  schedule,  and 
performance  by  asking  these  questions:  1)  How  much  will  the  project  finally  cost?;  2)  How 
long  will  the  project  eventually  take?;  and  3)  Will  this  product  perform  according  to 
specifications?  (Galway,  2004,  p.  ix).  Moreover,  his  approach  agrees  with  the  cost, 
schedule,  and  performance  categorization  of  risk  found  in  the  Risk  Management  Guide  for 
DoD  Acquisition  (USD[AT&Lj,  2006,  pp.  13-16). 

From  Qualitative  to  Quantitative  Risk  Analysis  Using  Existing  DoD  Guidance 

Could  the  current  Risk  Management  Guide  for  DoD  Acquisition  be  revised  to  include 
more  quantitative  risk  analysis?  The  answer  is  yes.  During  risk  assessment,  the  guide 
requires  the  analysis  of  future  root  causes  of  risk  as  to  probability  of  occurrence  and  severity 
or  consequence  if  that  future  root  cause  is  realized.  Simply  multiplying  the  probability  of  a 
risk  event  (expressed  as  a  decimal)  by  a  monetized  severity  of  loss  would  yield  a  monetary 
risk  number.  Adding  up  all  of  those  monetary  risk  numbers  for  all  elements  of  an  acquisition 
program  would  provide  an  informative,  albeit  somewhat  simplified  quantitative  risk 
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assessment.  There  are  more  sophisticated  approaches  to  quantifying  risk  that  could  be 
applied  to  risk  due  to  deficient  contractor  business  systems.  These  methods  include 
expected  value  of  information  (EVI)  decision  theory  and  value  at  risk  (VaR)  methods,  both  of 
which  consider  probability  distributions. 

Expected  Value  of  Information  (EVI) 

One  government  response  to  public  comments  related  to  the  risk  of  harm  and 
materiality  of  deficiencies  in  the  December  2010  draft  business  systems  rule  was  stated  as 
follows:  “In  most  cases,  the  financial  impact  of  a  system  deficiency  cannot  be  quantified 
because  the  system  produces  unreliable  information”  (Federal  Register,  2011,  p.  28859). 
Hubbard  (2007)  would  challenge  that  statement.  Hubbard  maintains  that  understanding 
quantitative  risk  is  the  key  to  understanding  how  to  value  information  (or,  by  implication, 
partial  information  or  unreliable  information). 

Using  decision  theory,  Hubbard  explains  an  approach  for  measuring  the  expected 
value  of  information  (Hubbard,  2007,  chapter  7).  His  basic  equation  is  as  follows: 

EVI  =  EOL  ( before  information )  —  EOL  ( after  information)  (1 ) 

where, 

EVI  =  Expected  Value  of  Information 

EOL  =  Expected  Opportunity  Loss  =  Chance  of  being  wrong  x  Cost  of  being  wrong 

(Hubbard,  2007,  p.  88) 

Hubbard  (2007)  contends  that  EOL  exists  because  of  uncertainty  that  clouds  the 
negative  consequences  of  our  decisions.  By  acquiring  more  information,  uncertainty  will  be 
reduced  and,  in  turn,  the  EOL  will  be  reduced.  Moreover,  the  Expected  Value  of  Perfect 
Information  (EVPI)  can  be  determined  when  uncertainty  is  eliminated  (Hubbard,  2007,  pp. 
89-89). 

Hubbard  (2007)  contends  that  EOL  exists  because  of  uncertainty  that  clouds  the 
negative  consequences  of  our  decisions.  By  acquiring  more  information,  uncertainty  will  be 
reduced  and,  in  turn,  the  EOL  will  be  reduced.  Moreover,  the  Expected  Value  of  Perfect 
Information  (EVPI)  can  be  determined  when  uncertainty  is  eliminated  (Hubbard,  2007,  pp. 
89-89). 

This  approach  might  be  useful  for  the  government  contracting  officer.  EVPI  might  be 
calculated  for  each  of  the  six  contractor  business  system,  assuming  that  the  information 
provided  is  flawless  so  that  the  decisions  made  from  that  information  are  never  wrong.  The 
sum  of  each  of  the  six  business  system  EVPIs  would  represent  the  total  dollar  value  to  the 
government  of  a  particular  contractor’s  business  systems. 

Value  at  Risk  (VaR) 

Jorion  (2007)  has  written  one  of  the  most  comprehensive  texts  on  how  banks  and 
other  financial  institutions  use  value  at  risk  (VaR)  to  quantify  potential  investment  losses.  He 
defines  VaR  as  “the  worst  loss  over  a  target  horizon  such  that  there  is  a  low,  prespecified 
probability  that  the  actual  loss  will  be  larger”  (Jorion,  2007,  p.  106).  Knowing  the  VaR  with  a 
high  degree  of  confidence  tells  the  firm  how  much  cash  and  capital  reserves  are  necessary 
to  cover  the  risk.  Mathematically,  Jorion’s  concept  of  VaR  looks  like  this: 

P(L  >  VaR)  <  1  -c  (2) 

where, 

P  =  Probability 
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L  =  Loss,  measured  as  a  positive  number 

VaR  =  Value  at  Risk,  reported  as  a  positive  number 

c  =  confidence  level,  expressed  as  a  decimal  (e.  g. ,  .99) 

(Jorion,  2007,  p.106) 

Jorion’s  approach  to  quantifying  downside  risk  has  five  steps:  (1)  Determine  the 
current  value  of  the  portfolio  (mark  position  to  market);  (2)  Measure  the  variability  of  the 
various  risk  factors,  such  as  changes  in  interest  rates,  fluctuations  in  foreign  exchange 
rates,  and  volatility  in  commodity  and  equity  prices;  (3)  Since  volatility  grows  with  the  square 
root  of  time,  set  the  time  horizon,  based  on  the  trading  days  over  which  the  assets  will  be 
held;  (4)  Select  the  desired  confidence  level;  and  (5)  calculate  the  potential  loss  (Jorion, 
2007,  p.  107).  In  its  simplest  form,  VaR  can  be  expressed  as  follows: 

VaR  =  Current  Market  Value  ($)  x  a  x  Jt] 252  x  a  (3) 

where, 

a  =  measured  variability  of  risk  factors,  expressed  as  a  percentage 

t  =  trading  days-,  252  =  trading  days  in  a  year 

a  =  confidence  level,  expressed  as  a  factor  (e.  g. ,  assuming  99%  confidence 
and  a  normal  distribution,  the  factor  would  be  2.33) 

From  this  basic  VaR  equation,  Jorion  (2007)  presents  three  models  for  calculating 
VaR.  The  local-valuation  method  determines  the  value  of  a  portfolio  once  and  uses 
mathematical  derivatives,  both  linear  and  quadratic,  to  measure  exposures  to  the  variability 
of  risk  (a)  (Jorion,  pp.  249-251).  The  full-valuation  method  reprices  a  financial  portfolio  over 
a  range  of  risk  scenarios,  using  tools  such  as  Monte  Carlo  simulation  (Jorion,  2007,  pp. 
251-257).  And,  the  historical  simulation  method  is  a  nonparametric  approach  that  “replays” 
the  tape  of  history  on  the  current  market  value  of  a  portfolio  (Jorion,  2007,  pp.  262-263). 

Hendricks  (1996)  has  evaluated  these  three  methods  of  calculating  VaR  using 
historical  financial  market  data.  He  concludes  that  all  of  these  methods  cover  the  risks  they 
are  intended  to  cover  and  that  the  risk  estimates  do  not  differ  much  in  average  size. 
Substantial  differences  between  the  methods  are  seen  when  comparing  methods  for  one 
trading  date;  however,  less  variation  is  seen  over  longer  periods  (Hendricks,  1996,  pp.55- 
56).  Hendricks’  method  of  validating  VaR  methods  using  historical  market  data  is  also 
instructive  for  this  research  into  the  risk  analysis  of  deficient  business  system.  Unless  the 
quantitative  risk  methods  can  be  verified  as  covering  the  government’s  risks  and  unless 
those  methods  can  be  used  by  the  contracting  officer,  they  are  of  little  use. 

Kaufman  and  Sougstad  (2007)  use  VaR  in  a  slightly  different  manner.  They  use  VaR 
as  a  constraint  function  to  help  make  better  decisions  on  risks  associated  within  a  portfolio 
of  information  technology  (IT)  service  contracts.  They  consider  the  risk  levels  of  various 
mixes  of  these  IT  contracts  (high,  medium,  and  low)  and  maximize  an  objective  function  for 
profitability,  subject  to  the  VaR  constraint.  This  approach  might  be  useful  for  the  government 
contracting  officer  when  selecting  a  portfolio  of  services  contractors  within  a  military 
contingency  operation,  while  keeping  the  overall  program  within  a  predetermined  VaR  to  the 
government. 

Operational  Value  at  Risk  (VaR) 

Jorion  (2007)  devotes  an  entire  chapter  to  Operational  Risk  Management,  an 
approach  to  quantifying  risk  using  methods  from  the  insurance  industry  and  from  VaR.  In  the 
financial  sector,  operational  risk  is  rooted  in  fraud  and  poor  governance.  Realized 
operational  risks  come  in  two  varieties.  High-frequency/low-value  risks  occur  mainly  from 
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daily  transaction  errors  and  poorly  implemented  internal  controls  that  do  not  catch  these 
errors.  Data  on  these  relatively  small  but  frequent  losses  is  easy  to  collect,  provided  that  the 
losses  are  reported.  On  the  other  hand,  low-frequency/high-value  risks  are  the  big  losses 
that  ultimately  bankrupt  an  institution.  Examples  include  the  corporate  lies  that  resulted  in 
the  Enron  scandal  in  2001  and  the  more  recent  subprime  mortgage  crisis  that  led  to  the 
bankruptcy  of  Lehman  Brothers  in  2008.  When  combined,  the  continuous  distribution  of  the 
severities  of  both  of  these  types  of  risk  losses  decreases  exponentially  with  the  size  of  the 
loss.  The  log-normal,  Weibull,  and  gamma  probability  distributions  are  often  used  to  model 
this  type  of  loss  distribution  (Jorion,  2007,  pp.  501,  511).  The  gamma  distribution  is  shown  in 
Figure  1 . 


Figure  1.  Gamma  Distribution 

(National  Institute  of  Standards  and  Technology,  2012) 

Jorion  (2007,  p.  497)  measures  operational  risk  using  a  4-step  approach:  (1)  Define 
risk  categories  such  as  processes,  people,  systems,  and  external  events;  (2)  Measure  risk 
factors  in  terms  of  loss  frequency;  (3)  Measure  exposure  in  terms  of  loss  severity;  and  (4) 
Calculate  risk,  either  operational  VaR  or  expected  loss.  The  probability  distribution  function 
of  loss,  used  in  the  calculation  of  either  operational  VaR  or  expected  loss,  is  determined 
using  this  integral: 

probability  distribution  function  of  loss  =  J  g{s  |  n  )  /(n)  dn  (4) 

where, 

g(s  |  n)  =  probability  distribution  function  of  the  sum  of  severity  variables 
s  =  loss  severity 
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n  =  loss  frequency  (number  of  occurences  of  loss  over  the  time  horizon ) 

/(n)  =  probability  distribution  function  of  the  loss  frequency  (n) 

From  Equation  4,  we  see  that  the  probability  distribution  function  of  the  aggregated 
loss  is  a  combination  of  the  loss  frequency  and  the  loss  severity  probability  distributions. 
Combining  the  frequency  and  severity  distributions  can  either  be  done  manually  by 
systematically  tabulating  all  possible  combinations  (the  process  is  termed  convolution)  or  by 
the  integral,  as  shown  in  Equation  4  (Jorion,  2007,  pp.  498-501). 


Research  Methodology 

This  initial  research  effort  addressed  only  EVM  systems,  one  of  the  six  contractor 
business  systems  identified  in  section  893  of  the  National  Defense  Authorization  Act  for 
Fiscal  Year  2011.  Data  was  extracted  from  contract  performance  reports  (CPR)  in  the 
Defense  Cost  and  Resource  Center  (DCARC)  EVM  Central  Repository.  The  repository 
provides  and  supports  the  centralized  reporting,  collection,  and  distribution  for  key 
acquisition  EVM  data,  such  as  CPRs,  Contract  Funds  Status  Report  (CFSR),  and  the 
Integrated  Master  Schedule  (IMS)  for  Acquisition  Category  1C  &  ID  (Major  Defense 
Acquisition  Programs)  as  well  as  ACAT  1A  (Major  Automated  Information  System) 
programs. 

When  monthly  CPRs  are  submitted  by  the  defense  contractor,  government  analysts 
must  check  to  ensure  that  the  data  makes  sense.  If  errors  are  present  and  remain 
undetected,  the  data  could  result  in  poor  decisions  by  both  the  contractor  and  the 
government.  This  is  called  a  validity  check  of  the  CPR  data  and  the  check  ties  most  directly 
to  the  accounting  related  group  of  EVM  system  guidelines  (16,  17,  18,  19,  20,  21 ;  a 
description  of  the  EVM  system  guidelines  is  at  Appendix  A). 

The  DCARC  provides,  free  of  charge  to  both  government  and  contractor  EVMS 
analysts,  CPR  file  viewer  software  that  includes  a  tool  designed  to  perform  validity  checks 
on  the  EVM  data  contained  in  the  CPR.  There  are  a  total  of  10  validity  checks  included  in 
the  software.  These  validity  checks  have  been  endorsed  by  the  Government  Accountability 
Office  Cost  Estimating  and  Assessment  Guide  (GAO-09-3SP;  GAO,  2009)  and  the  Society 
of  Cost  Estimating  and  Analysis  Cost  Estimating  Body  of  Knowledge  (SCEA  CEBoK). 

Two  of  the  validity  checks  that  provide  a  definitive  measure  of  data  validity  are  as 
follows:  (1)  the  cumulative  Budgeted  Cost  of  Work  Performed  (BCWP)  has  value  with  no 
corresponding  cumulative  Actual  Cost  of  Work  Performed  (ACWP);  and  (2)  cumulative 
ACWP  has  value  with  no  corresponding  cumulative  BCWP.  The  first  validity  check  indicates 
that  budgeted  and  authorized  work  performed  in  a  particular  Work  Breakdown  Structure 
(WBS)  element  was  recorded  in  the  EVM  system,  but  no  labor  or  material  costs  were 
recorded  in  the  company’s  accounting  system  for  that  work.  The  effect  of  this  error  is  to 
understate  the  actual  cost  of  work  that  has  been  performed.  The  second  validity  check 
reveals  that  actual  labor  and  material  costs  (ACWP)  were  recorded  against  a  particular  WBS 
in  the  accounting  system,  but  no  earned  value  (BCWP)  was  credited  in  the  EVM  system  of 
the  company.  The  effect  of  this  error  is  to  overstate  the  actual  cost  of  work  that  has  been 
performed. 

Three  system  development  efforts,  from  three  different  defense  contractors,  were 
selected  from  the  DCARC  EVM  Central  Repository.  For  each  of  these  systems,  three 
separate  monthly  CPRs  were  selected.  The  two  validity  checks  discussed  in  this  section 
were  performed  on  each  of  these  nine  CPRs. 
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Results  Analysis  and  Conclusion 

Loss  distributions  were  analyzed  using  Risk  Solver  Platform  by  Frontline  Systems.  In 
most  cases,  the  loss  distributions  were  best  fit  by  the  gamma  distribution.  Thus,  Jorion’s 
(2007)  operational  VaR  method  could  be  used  to  estimate  the  risk  to  the  government  of 
deficient  information  from  contractor  EVM  systems. 

Use  of  the  VaR  model  would  benefit  the  government  contracting  officer  by  providing 
a  defensible  risk  value  as  the  basis  for  withholding  contractor  payments. 

Recommendations  for  Further  Research 

Future  research  should  define  a  “significant  deficiency”  in  an  EVM  system  and  what 
it  means  for  that  significant  deficiency  to  materially  affect  the  ability  to  rely  upon  information 
produced  by  the  EVM  system. 

Future  research  should  map  significant  deficiencies  to  risk  of  harm  to  the 
government  (future  root  causes  of  risk  with  probability  and  consequence).  Risk  of  harm  may 
be  outside  the  contract  (e.g.,  new  replacement  aircraft  carrier  will  be  delivered  late,  requiring 
old  aircraft  carrier  to  be  retained  in  service  with  added  operating  and  maintenance  costs). 
Risk  of  harm  may  include  the  value  of  not  having  a  needed  combat  capability  (i.e.,  liquidated 
damages).  Moreover,  risk  of  harm  may  include  such  things  as  supply  chain  and  industrial 
base  risks.  In  addition,  future  research  should  take  a  look  at  current  EVM  system  metrics 
that  are  already  collected  on  a  regular  basis  and  focus  on  the  impact  (if  any)  those  metrics 
have  on  risk  of  harm. 

Future  research  should  include  validation  of  the  operational  VaR  model  and  the 
simplification  of  that  model  into  a  template  that  can  be  used  by  contracting  officers. 

Research  should  also  be  conducted  into  changes  needed  to  the  DCMA  contractor  business 
systems  instruction,  EVM  system  compliance  review  instruction,  and  EVM  system  standard 
surveillance  instruction  as  a  result  of  using  operational  VaR. 

Once  the  operational  VaR  method  is  devised  for  significant  materials  deficiencies  in 
EMV  systems,  apply  operational  VaR  (or  another  more  suitable  quantitative  method)  to  the 
other  five  business  systems  covered  by  DFARS  (2011)  252.234.7005,  being  careful  to  first 
define  significant  deficiency,  materially  affect,  and  risks  of  harm  for  each  of  those  business 
systems,  separately. 

Additional  research  proposed  by  the  DCMA  is  found  at  Appendix  B. 

References 

Brodsky,  R.  (2010,  December  6).  Defense  revises  rule  on  contractor  business  systems.  Retrieved 
from  Government  Executive  website:  http://www.govexec.com 

Capaccio,  T.  (2012,  February  28).  Pentagon  to  withhold  Lockheed  F-35  payments  over  tracking. 
Retrieved  from  Bloomberg  News  website:  http://bloomberg.com 

Commission  on  Wartime  Contracting  in  Afghanistan  and  Iraq.  (2009,  June  1 0).  At  what  cost? 

Contingency  contracting  in  Iraq  and  Afghanistan  (Interim  Report  to  Congress).  Retrieved  from 
http://www.wartimecontracting.gov 

Commission  on  Wartime  Contracting  in  Afghanistan  and  Iraq.  (2009,  September  21).  Defense 
agencies  must  improve  their  oversight  of  contractor  business  systems  to  reduce  fraud,  waste, 
and  abuse  (Special  Report  1  on  Contractor  Business  Systems).  Retrieved  from 
http://www.wartimecontracting.gov 

Commission  on  Wartime  Contracting  in  Afghanistan  and  Iraq.  (2011,  August  31).  Transforming 

wartime  contracting:  Controlling  costs,  reducing  risks  (Final  Report  to  Congress).  Retrieved  from 
http://www.wartimecontracting.gov 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-408- 


Defense  Contract  Management  Agency  (DCMA).  (2011,  June).  Earned  value  management  system 
(EVMS)  compliance  review  instruction  (CRI).  Retrieved  from  DCMA  guidebook  website: 
http://guidebook.dcma.mil 

Defense  Contract  Management  Agency  (DCMA).  (2012a,  January).  Contractor  business  systems. 
Retrieved  from  DCMA  guidebook  website,  instruction  folder  number  308: 
http://guidebook.dcma.mil 

Defense  Contract  Management  Agency  (DCMA).  (2012b,  February).  Earned  value  management 
system  (EVMS)  standard  surveillance  instruction  (SSI).  Retrieved  from  DCMA  guidebook 
website:  http://guidebook.dcma.mil 

Defense  Federal  Acquisition  Regulation  Supplement  (DFARS),  48  C.F.R.  ch.  2  (2011).  Retrieved 
from  http://www.acq.osd.mil/dpap/dars/dfarspgi/current/index.html 

DoD.  (2012,  January  10).  Defense  acquisition  guidebook.  Retrieved  from  https://dag.dau.mil 

Federal  Register.  (2011,  May  18).  Defense  federal  acquisition  regulation  supplement:  Business 
systems — Definition  and  administration;  interim  rule,  76(96),  28858-28879. 

Fleming,  Q.  W.,  &  Koppelmann,  J.  M.  (2005).  Earned  value  project  management  (3rd  ed.).  Newtown 
Square,  PA:  Project  Management  Institute. 

Galway,  L.  (2004,  February).  Quantitative  risk  analysis  for  project  management:  A  critical  review 
(RAND  working  paper  WR-1 12-RC).  Retrieved  from 
http://www.rand.org/pubs/working_papers/2004/RAND_WR1 12.pdf 

GAO.  (2009,  March).  GAO  cost  estimating  and  assessment  guide:  Best  practices  for  developing  and 
managing  capital  program  costs  (GAO-09-3SP).  Retrieved  from  http://www.gao.gov 

Hendricks,  D.  (1996,  April).  Evaluation  of  value-at-risk  models  using  historical  data.  FRBNY 
Economic  Policy  Review,  39-69. 

Hubbard,  D.  W.  (2007).  How  to  measure  anything:  Finding  the  value  of  “intangibles”  in  business. 
Hoboken,  NJ:  Wiley. 

Jorion,  P.  (2007).  Value  at  risk:  The  new  benchmark  for  managing  financial  risk  (3rd  ed.).  New  York, 
NY:  McGraw-Hill. 

Kaufman,  R.,  &  Sougstad,  R.  (2007).  Value-at-risk  in  IT  services  contracts.  In  Proceedings  of  the  40th 
Hawaii  International  Conference  on  System  Sciences — 2007.  Institute  of  Electrical  and 
Electronics  Engineers. 

National  Defense  Authorization  Act  for  Fiscal  Year  201 1 ,  §  893  (201 1 ). 

National  Institute  of  Standards  and  Technology.  (2012,  April).  NIST/SEMATECH  e-handbook  of 
statistical  methods.  Retrieved  from  http://www.itl.nist.gov/div898/handbook/ 

USD(AT&L).  (2006,  August).  Risk  management  guide  for  DoD  acquisition.  Washington,  DC:  Author. 

Appendix  A.  ANSI/EIA-748  Earned  Value  Management  System  (EVMS) 
Guidelines  (Fleming  &  Koppelmann,  2005,  pp.  191-214) 

ANSI/EIA-748  GUIDELINES 

ORGANIZATION _ 

Criterion  1.  Define  the  authorized  work  elements  for  the  agency.  A  WBS,  tailored  for 
effective  internal  management  control,  is  commonly  used  in  this  process. _ 

Criterion  2.  Identify  the  organizational  structure  including  the  major  contractors  responsible 
for  accomplishing  the  authorized  work,  and  define  the  organizational  elements  in  which  work 
will  be  planned  and  controlled. _ 

Criterion  3.  Provide  for  the  integration  of  the  agency’s  planning,  scheduling,  budgeting,  work 
authorization  and  cost  accumulation  processes  with  each  other,  the  WBS,  and  the  OBS. 
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Criterion  4.  Identify  the  organization  or  function  responsible  for  controlling  overhead  (indirect 
costs). 

Criterion  5.  Provide  for  integration  of  the  WBS  and  the  organizational  structure  in  a  manner 
that  permits  cost  and  schedule  performance  measurement  by  elements  of  either  or  both 
structures  as  needed. 

PLANNING,  SCHEDULING,  AND  BUDGETING 

Criterion  6.  Schedule  the  authorized  work  in  a  manner  that  describes  the  sequence  of  work 
and  identifies  significant  task  interdependencies  required  to  meet  the  requirements  of  the 
program. 

Criterion  7.  Identify  physical  products,  milestones,  technical  performance  goals,  or  other 
indicators  that  will  be  used  to  measure  progress. 

Criterion  8.  Establish  and  maintain  a  time-phased  budget  baseline,  at  the  control  account 
level,  against  which  performance  can  be  measured.  Budget  for  far-term  efforts  may  be  held 
in  higher-level  accounts  until  an  appropriate  time  for  allocation  at  the  control  account  level. 
Initial  budgets  established  for  performance  measurement  will  be  based  on  either  internal 
management  goals  or  the  external  customer-negotiated  target  cost  including  estimates  for 
authorized  but  undefined  work. 

Criterion  9.  Establish  budgets  for  authorized  work  with  identification  of  significant  cost 
elements  (labor,  material,  etc.)  as  needed  for  internal  management  and  for  control  of 
contractors. 

Criterion  10.  To  the  extent  it  is  practical  to  identify  the  authorized  work  in  discrete  work 
packages,  establish  budgets  for  this  work  in  terms  of  dollars,  hours,  or  other  measurable 
units.  Where  the  entire  control  account  is  not  subdivided  into  work  packages,  identify  the  far 
term  effort  in  larger  planning  packages  for  budget  and  scheduling  purposes. 

Criterion  11.  Provide  that  the  sum  of  all  work  package  budgets  plus  planning  package 
budgets  within  a  control  account  equals  the  control  account  budget. 

Criterion  12.  Identify  and  control  level  of  effort  activity  by  time-phased  budgets  established 
for  this  purpose.  Only  that  effort  which  is  immeasurable  or  for  which  measurement  is 
impractical  may  be  classified  as  level  of  effort. 

Criterion  13.  Establish  overhead  budgets  for  each  significant  organizational  component  of 
the  company  for  expenses  that  will  become  indirect  costs.  Reflect  in  the  budgets,  at  the 
appropriate  level,  the  amounts  in  overhead  pools  that  are  planned  to  be  allocated  as  indirect 
costs. 

Criterion  14.  Identify  management  reserves  and  undistributed  budget. 

Criterion  15.  Provide  that  the  allocated  budget  is  reconciled  with  the  sum  of  all  internal 
budgets  and  management  reserves. 

ACCOUNTING 

Criterion  16.  Record  direct  costs  in  a  manner  consistent  with  the  budgets  in  a  formal  system 
controlled  by  the  general  books  of  account. 

Criterion  17.  Summarize  direct  costs  from  control  accounts  into  the  WBS  without  allocation 
of  a  single  control  account  to  two  or  more  WBS  elements. 

Criterion  18.  Summarize  direct  costs  from  the  control  accounts  into  the  agency’s 
organizational  elements  without  allocation  of  a  single  control  account  to  two  or  more 
organizational  elements. 

Criterion  19.  Record  all  indirect  costs  that  will  be  allocated  to  the  agency. 

Criterion  20.  Identify  unit  costs,  equivalent  units  costs,  or  lot  costs  when  needed. 

Criterion  21.  For  EVMS,  the  material  accounting  system  will  provide  for: 

•  Accurate  cost  accumulation  and  assignment  of  costs  to  control  accounts  in  a  manner 
consistent  with  the  budgets  using  recognized,  acceptable,  costing  techniques. 

•  Cost  performance  measurement  at  the  point  in  time  most  suitable  for  the  category  of 
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material  involved,  but  no  earlier  than  the  time  of  progress  payments  or  actual  receipt 
of  material. 

•  Full  accountability  of  all  material  purchased  including  the  residual  inventory. 

MANAGERIAL  ANALYSIS  &  REPORTING 

Criterion  22.  At  least  on  a  monthly  basis,  generate  the  following  information  at  the  control 
account  and  other  levels  as  necessary  for  management  control  using  actual  cost  data  from, 
or  reconcilable  with,  the  accounting  system: 

•  Comparison  of  the  amount  of  planned  budget  and  the  amount  of  budget  earned  for 
work  accomplished.  This  comparison  provides  the  schedule  variance. 

•  Comparison  of  the  amount  of  the  budget  earned  with  the  actual  (applied  where 
appropriate)  direct  costs  for  the  same  work.  This  comparison  provides  the  cost 
variance. 

Criterion  23.  Identify,  at  least  monthly,  the  EV  Variance  between  both  planned  and  actual 
schedule  performance  and  planned  and  actual  cost  performance,  and  provide  the  reasons 
for  the  variances  in  the  detail  needed  by  management. 

Criterion  24.  Identify  budgeted  and  applied  (or  actual)  indirect  costs  at  the  level  and 
frequency  needed  by  management  for  effective  control,  along  with  the  reasons  for  any 
significant  variances. 

Criterion  25.  Summarize  the  data  elements  and  associated  variances  through  the 
organization  and/or  WBS  to  support  management  needs  and  any  customer  reporting 
specified  in  the  contract. 

Criterion  26.  Implement  managerial  actions  taken  as  the  result  of  earned  value  information. 

Criterion  27.  Develop  revised  EAC  based  on  performance  to  date,  commitment  values  for 
material,  and  estimates  of  future  conditions.  Compare  this  information  with  the  performance 
measurement  baseline  to  identify  variances  at  completion  important  to  company 
management  and  any  applicable  customer  reporting  requirements  including  statements  of 
funding  requirements. 

BASELINE  MANAGEMENT 

Criterion  28.  Incorporate  authorized  changes  in  a  timely  manner,  recording  the  effects  of 
such  changes  in  budgets  and  schedules.  In  the  directed  effort  prior  to  negotiation  of  a 
change,  base  such  revisions  on  the  amount  estimated  and  budgeted  to  the  organizations. 

Criterion  29.  Reconcile  current  budgets  to  prior  budgets  in  terms  of  changes  to  the 
authorized  work  and  internal  replanning  in  the  detail  needed  by  management  for  effective 
control. 

Criterion  30.  Control  retroactive  changes  to  records  pertaining  to  work  performed  that  would 
change  previously  reported  amounts  for  actual  costs,  earned  value,  or  budgets.  Adjustments 
should  be  made  only  for  correction  of  errors,  routine  accounting  adjustments,  effects  of 
customer  or  management  directed  changes,  or  to  improve  the  baseline  integrity  and 
accuracy  of  performance  measurement  data. 

Criterion  31.  Prevent  revisions  to  the  agency  budget  except  for  authorized  changes. 

Criterion  32.  Document  changes  to  the  performance  measurement  baseline. 

Appendix  B.  Proposed  Research  on  Significant  Material  Deficiency,  Prepared 
by  DCMA,  March  28,  2012 

QUANTITATIVE  METHODS  FOR  ESTIMATING  SIGNIFICANT  MATERIAL 

DEFICIENCY 


References: 

(a)  DFARS:  Business  systems,  Interim  rule,  Federal  Register,  Vol.  76,  No.  96,  18 
May  2011. 
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(b)  National  Defense  Authorization  Act  for  FY  201 1 ,  sect.  893,  Contractor  Business 
Systems. 

(c)  ANSI/EIA  748C  EVMS  Guidelines 

(d)  DCMA  EVMS  Compliance  Review  Instruction 

(e)  DCMA  Contractor  Business  Systems  Instruction 

1.  BACKGROUND — Statutory  and  regulatory  guidance  in  references  (a)  and  (b)  identify  six 
(6)  contractor  business  systems  that  may,  when  providing  inaccurate  and  unreliable  data, 
create  a  “risk  of  harm”  to  the  government.  When  that  risk  is  substantiated  and  validated  by 
the  discovery  of  significant  deficiencies  in  the  contractor’s  business  system,  reference  (a) 
authorizes  the  government  to  withhold  payment  to  the  contractor  to  mitigate  the  risk.  In 
many  cases,  the  financial  impact  and  materiality  of  the  deficiency  is  difficult  to  quantify,  since 
it  is  the  inaccurate  and  unreliable  data  produced  by  the  business  system  that  is  (in  fact)  the 
deficiency.  Although  six  business  systems  are  covered  by  references  (a)  and  (b),  the  initial 
focus  on  this  research  is  the  earned  value  management  system  (EVMS).  Methods, 
practices,  and  guidelines  for  establishing  EVMS  deficiencies  are  fairly  mature.  Reference  (c) 
guidelines  are  explicitly  cited  as  “high  risk”  guidelines  (GL: 

1,3,6,7,8,9,10,12,16,21,23,26,27,28,30,32).  A  significant  deficiency  in  any  one  of  these 
guidelines  constitutes  EVMS  non-compliance.  In  any  case,  the  standard  for  withholding 
payments  to  the  contractor  is  commensurate  with  the  ($)  risk  to  the  government.  It  is  the 
causal  relationship  between  risk  and  system  deficiency  and  the  calculation  of  a  risk  value  (in 
$dollars)  that  is  the  core  issue.  A  consistent,  repeatable  and  quantitative  method  is  needed 
to  fairly  determine  if  payment  withholds  of  5%  are  necessary  to  mitigate  the  government’s 
risk. 

2.  RESEARCH  TOPICS/QUESTIONS — The  following  research  topics  are  suggested: 

a.  Similar  risk  management  problems  exist  in  industry,  including  insurance,  banking, 
and  investment  portfolio  management.  In  the  insurance  industry,  actuarial  data  is  applied  to 
model  risk  and  risk  capital  reserves  are  “held”  to  cover  the  expected  values  for  claims.  In 
banking,  the  banks  hold  cash  reserves  to  maintain  liquidity  and  cover  the  expected  values  of 
depositor  withdrawals,  based  on  regulatory  requirements  for  capital  (risk)  reserves.  In 
investment  portfolios,  managers  evaluate  value  at  risk  across  the  portfolio  and  adjust 
holdings  as  necessary  to  maintain  the  risk  value  below  acceptable  levels.  In  all  three  of 
these  examples,  the  scenario  manages  risk  in  an  environment  that  is  characterized  by 
probabilistic  behavior,  expected  ($)  values  and  risk  management  objectives  that  are 
governed  by  regulation  and  statutory  law.  Research  questions  are  as  follows: 

i.  Can  any  of  these  risk  management  models  (or  others)  be  extrapolated  to  the 
business  system  rule  implementation? 

ii.  What  quantitative  methods  can  be  developed  to  quantify  material  financial 
risk  to  government  when  a  business  system  produces  “unreliable  and 
inaccurate  data”? 

iii.  How  do  other  regulatory  activities  manage  risk  that  is  caused  by  business 
system  deficiencies? 

b.  EVMS  are  “critical  guidelines”  (cited  above).  When  a  significant  deficiency  exists  in 
one  of  these  guidelines  the  EVMS  is  non-compliant  and  will  be  disapproved  by  the 
administrative  contracting  officer  (ACO).  A  significant  deficiency  in  a  non-high  risk  guideline 
may  result  in  disapproval  at  the  discretion  of  the  ACO.  The  following  research  questions, 
specific  to  EVMS  apply: 

i.  Is  there  any  rank  or  natural  order  to  the  potential  severity  of  the  deficiency 
posed  by  these  guidelines?  To  what  degree  is  there  inter-dependence  or 
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causality  across  these  guidelines?  Can  they  be  group  or  “binned”  WRT 
causality  of  risk? 

ii.  What  quantitative  method(s)  can  be  used  to  calculate  risk  value  WRT  non- 
compliance  of  critical  guidelines,  or  non-critical  guidelines? 

iii.  What  quantitative  definition  of  “significant  deficiency”  is  applicable? 

iv.  Is  it  possible  to  develop  a  deterministic  rule  set  that  yields  a  consistent  and 
repeatable  finding  of  significant  deficiency? 

v.  What  should  be  the  relationship  of  risk  value  calculations  and  findings  of 
EVMS  (GL)  non-compliance  with 

a.  Probability  of  error, 

b.  Magnitude  of  errors,  and 

c.  Adverse  impact  of  errors 

c.  Other  business  systems  are  also  within  the  scope  of  references  (a)  and  (b);  the 
following  questions  apply: 

i.  Can  an  extensible  method  be  developed  for  all  business  systems  WRT  the 
calculation  of  risk  value? 

ii.  What  is  the  standard  of  compliance  that  applies  to  the  non-EVMS  business 
systems?  Do  these  standards  have  clearly  defined  “compliance”  criteria? 

d.  Instead  of  “measuring”  business  system  compliance  with  various  guidelines  and 
standards,  is  there  a  better  approach  to  risk  management,  when  there  is  potential  harm  to 
the  government? 
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Research  Issue 


Request  from  Director  of  the  Earned  Value  Management  Division  of  the 
Defense  Contract  Management  Agency  (DCMA) 

Asked  for  assistance  in  developing  a  method  that  Administrative  Contracting 
Officers  (ACO)  can  use  to  assess  risk  associated  with  non-compliant 
contractor  business  systems 

A  recent  rule  change  in  the  Defense  Supplement  to  the  Federal  Acquisition 
Regulation  (DFARS)  permits  an  ACO  to  withhold  up  to  10%  of  contract 
payments  for  a  "significant  deficiency"  in  a  contractor's  business  system  that 
creates  risk  to  the  government. 

The  research  issue  is  how  to  objectively  and  quantitatively  portray  that  risk  in 
a  way  that  supports  a  monetary  withhold  decision  and  can  withstand  push- 
back  (to  include  litigation)  from  the  defense  contractor. 


Definition  of  Terms 
Contractor  Business  Systems 
Section  893,  NDAA  for  FY2011 


Term 

Definition 

Contractor 

Business 

System 

•  accounting  system 

•  estimating  system 

•  purchasing  system 

•  earned  value  management  system 

•  material  management  and  accounting  system 

•  property  management  system 

Covered 

Contractor 

A  contractor  that  is  subject  to  the  cost  accounting  standards  under  section  26  of 
the  Office  of  Federal  Procurement  Policy  Act  (41  U.S.C.  422) 

Covered 

Contract 

•  cost-reimbursement  contract 

•  incentive-type  contract 

•  time-and-materials  contract 

•  labor-hour  contract 

Significant 

Deficiency 


.  .shortcoming  in  the  system  that  materially  affects  the  ability  of  officials  of  the 
Department  of  Defense  and  the  contractor  to  rely  upon  information  produced  by 
the  system  that  is  needed  for  management  purposes.” 


Feedback  on  December  2010  Draft  Rule 


The  rule  fails  to  offer  any  guidance  to  the  contracting 
officer  for  describing  a  "significant  deficiency." 

There  is  the  potential  for  inconsistent  application  of 
business  system  criteria  when  determining  a 
"significant  deficiency." 

The  Defense  Contract  Management  Agency  (DCMA) 
and  the  Defense  Contract  Audit  Agency  (DCAA)  are 
under-resourced  to  implement  the  rule. 


Constructing  the  Loss  Distribution* 


0.6 

0.4 

0.2 

0 


Frequency  Distribution 


Severity  Distribution 


o 


$1,000  $10,000  $100,000 


0.6 

0.4 

0.2 

0 


Loss  Distribution 


Expected  Loss 
($9,520) 

Value  at  Risk 

- > 

($58,480) 

- > 

■  ■  ■ 

1 _ ■ _ . 

0  1  2  10  11  20  100  101  110  200 

Loss  ($000) 


*Notional,  see  Jorion  (2007),  Value  at  Risk,  Chapter  19,  Operational  Risk 
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Earned  Value  Management 
32  Guidelines 


1.  Define  the  Project  Work  Scope 

2.  Define  the  Project  Organization 

3.  Integrate  Subsidiary  Processes 

4.  Identify  Overhead  Management 

5.  Integrate  WBS/OBS  to  Create  Control 
Accounts 

6.  Schedule  with  network  logic 

7.  Set  measurement  indicators 

8.  Establish  the  CBB/PMB 

9.  Budget  by  cost  elements 

10.  Create  work/planning  packages 

11.  Sum  WP/PP  budgets  to  the  Control 
Account 

12.  Level  of  Effort  planning 

13.  Set  overhead  budgets 

14.  Identify  Management  Reserve  and 
Undistributed  Budget 

15.  Reconcile  CBB  to  target  values 


Accounting  Processes 

16.  Record  direct  costs 

17.  Summarize  direct  costs  by  WBS  elements 

18.  Summarize  direct  costs  by  OBS  elements 

19.  Record/allocate  indirect  costs 

20.  Identify  unit  and  lot  costs  as  needed 

21.  Track  and  report  material  costs/quantities 

22.  Calculate  Schedule  Variance  &  Cost 
Variance 

23.  Identify  significant  variances  for  analysis 

24.  Analyze  indirect  Cost  Variance 

25.  Summarize  information  for  management 

26.  Implement  corrective  actions 

27.  Revise  Estimate  At  Completion 

28.  Incorporate  changes  in  a  timely  manner 

29.  Reconcile  current  to  prior  budgets 

30.  Control  retroactive  changes 

31.  Prevent  unauthorized  revisions 

32.  Document  PMB  changes 


Source:  Defense  Acquisition  University,  BCF-262  Card  EVMS  Guidelines 
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Contract  Performance  Report 
Validity  Checks 


Definitive 

GAO  Cost  Estimating  and  Assessment  Guide*  Measure? 


n egat i  ve  va  lues  for  ACW  P,  B  AC,  BCW  P .  BCWS,  o r  EAC 

Y 

u  n  usu  a  1  ly  la  rgE  p  e  rform  3  nee  swi  ngs  |  BCW  P ;  fro nn  m  o  nt  h  t  □  month 

H 

EO-V  F  a  n  d  BCWS  d  at  a  wit h  no  lorrES  ponding  ACW  P 

Y 

BCWP  with  no  BCWS 

Y 

San  ity  Checks 
(Anomalies) 

BCWP  with  no  ACW P 

Y 

ACW P  with  no  BCWP 

p.  257 

Y 

In  ions  ist  e  ncy  b  etw  e  e  n  EAC  3  n  d  B  AC,  such  as  n  o  EAC  b  ut  3  n  EAC  3  n  d  vie  e  ve  rz  3 

Y 

BCWP  or  BCWS  exceeds  BAC 

Y 

ACW  P  t  h  at  is  w  ay  a  b ove  orbs  low  t  h  a  p  1  a  n  n  Ed  va  lue 

H 

ACW P  eke e Eds  EAC 

Y 

*Similar  definitive  sanity  checks  are  found  in  the  SCEA  CEBoK,  p.  57 


Source:  CPR  File  Viewer  Users  Guide,  Version  1.0.2.20,  p.  30 
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Two  Validity  Checks  Selected 


BCWP  with  no  ACWP 

•  Indicates  that  budgeted  and 
authorized  work  performed  in 
a  particular  Work  Breakdown 
Structure  (WBS)  element  was 
recorded  in  EVM  system,  but 
no  labor  or  material  costs 
were  recorded  in  company's 
accounting  system  for  that 
work. 

•  The  effect  of  this  error  is  to 
understate  the  actual  cost  of 


ACWP  with  no  BCWP 

•  Reveals  that  actual  labor  and 
material  costs  (ACWP)  were 
recorded  against  a  particular 
WBS  in  the  accounting  system, 
but  no  earned  value  (BCWP) 
was  credited  in  the  EVM 
system  of  the  company. 

•  The  effect  of  this  error  is  to 
overstate  the  actual  cost  of 
work  that  has  been 
performed. 


work  that  has  been 
performed. 
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Program 

CPR  Date 

BCWP 

w/no 

ACWP 

Research  Results 

Dollar  %  of  WBS  ACWP 

Value  of  elements  w/no 

Errors  w/errors  BCWP 

Dollar 
Value  of 

Errors 

%  of  WBS 

elements 

w/errors 

Number  of 

WBS 

Elements 

1 

Feb-12 

6 

47443 

0.01 

13 

2359780 

0.02 

719 

1 

Nov-11 

7 

80043 

0.01 

10 

25000 

0.01 

719 

1 

Aug-11 

9 

114269 

0.01 

10 

33269 

0.01 

716 

2 

Feb-12 

30 

175295 

0.01 

324 

5722147 

0.15 

2216 

2 

Dec-11 

40 

272495 

0.02 

242 

6004275 

0.12 

2012 

2 

Aug-11 

24 

445108 

0.01 

217 

15931175 

0.13 

1635 

3 

Jan-12 

82 

1228323 

0.10 

30 

58140 

0.04 

820 

3 

Oct-11 

84 

1363864 

0.11 

22 

90663 

0.03 

779 

3 

Jul-11 

49 

1086391 

0.07 

11 

35545 

0.02 

710 
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Research  Results 


-60.36571615 
- 1 5.00%  I - r 


0.007 


0.006 


0.005 


g  0.004 

w 


0.003 


Qj 


75  0.002- 


355.1095939 


190.00% 


5.00% 


er  Platform 


or  E  do  cation 

ci at  Use 


or  qomm^r 

liii£  System  s,  Jnc 


+ 


-400  -200 


600 


H 


800  1000  1200 
(Valu  es  i  n  Thousands) 


Gamma  Loss  Distribution,  Program  2,  CPR  Aug  11,  ACWP  with  no  BCWP 
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Research  Conclusions 


•  Distributions  were  best  fit  by  Exponential  (2),  Gamma 
(4),  Weibull  (5),  and  LogLogistics  (6)  loss  distributions. 

•  An  Operational  Value  at  Risk  (VaR)  method  (Jorion,  2007) 
could  be  used  to  estimate  the  risk  to  the  government  of 
deficient  information  from  contractor  EVM  systems. 

•  Use  of  the  VaR  model  would  benefit  the  government 
contracting  officer  by  providing  a  more  defensible  risk 
value  as  the  basis  for  withholding  contractor  payments. 
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Future  Research 


Research  Topic  1 

Similar  risk  management  problems  exist  in  industry,  including:  insurance,  banking, 
and  investment  portfolio  management.  In  the  insurance  industry,  actuarial  data  is 
applied  to  model  risk  and  risk  capital  reserves  are  maintained  to  cover  the 
expected  values  for  claims.  Banks  hold  cash  reserves  to  maintain  liquidity  and 
cover  the  expected  values  of  depositor  withdrawals,  based  on  regulatory 
requirements  for  capital  (risk)  reserves.  Investment  portfolio  managers  evaluate 
value  at  risk  across  the  portfolio  and  adjust  holdings  as  necessary  to  maintain  the 
risk  value  below  acceptable  levels.  In  all  three  of  these  examples  the  risk 
environment  is  characterized  by  probabilistic  behavior,  expected  dollar  values,  and 
risk  management  objectives  that  are  governed  by  regulation  and  statutory  law. 

•  Can  any  of  these  risk  management  models  (or  others)  be  extrapolated  to  the 
business  system  rule  implementation? 

•  What  quantitative  methods  can  be  developed  to  quantify  material  financial  risk  to 
government  when  a  business  system  produces  "unreliable  and  inaccurate  data"? 

•  How  do  other  regulatory  activities  mange  risk  that  is  caused  by  business  system 
deficiencies? 
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Future  Research 


Research  Topic  2 

Under  ANSI/EIA-748,  Earned  Value  Management  Systems  (EVMS)  must  comply 
with  32  guidelines.  When  a  significant  deficiency  exists  in  one  of  the  critical 
guidelines  the  EVMS  is  non-compliant  and  will  be  disapproved  by  the 
Administrative  Contracting  Officer.  The  following  research  questions  apply 
specifically  to  EVMS: 

•  Is  there  any  rank  or  natural  order  to  the  potential  severity  of  the  deficiency  posed 
by  these  guidelines?  To  what  degree  is  there  inter-dependence  or  causality  across 
these  guidelines?  Can  they  be  group  with  respect  to  causality  of  risk? 

•  What  quantitative  method(s)  can  be  used  to  calculate  risk  value  with  respect  to 
non-compliance  of  critical  guidelines;  or  non-critical  guidelines? 

•  What  quantitative  definition  of  "significant  deficiency"  is  applicable? 

•  Is  it  possible  to  develop  a  deterministic  a  rule  set  that  yields  a  consistent  and 
repeatable  finding  of  significant  deficiency? 

•  What  should  be  the  relationship  of  risk  value  calculations  and  findings  of  EVMS 
non-compliance,  with:  1)  probability  of  error;  2)  magnitude  of  errors;  and,  3) 
adverse  impact  of  errors? 
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Future  Research 


Research  Topic  3 

Other  business  systems  (accounting,  estimating,  material  management  and 
accounting,  purchasing,  and  property  management)  are  also  within  the  scope  of 
the  contractor  business  system  rule  and  the  following  questions  apply: 

•  What  is  the  standard  of  compliance  that  applies  to  the  non-EVMS  business 
systems?  Do  these  standards  have  clearly  defined  compliance  criteria? 

•  Can  an  extensible  method  be  developed  for  all  business  systems  with  respect  to 
the  calculation  of  risk  value? 


Research  Topic  4 

Instead  of  measuring  business  system  compliance  with  various  guidelines  and 
standards,  is  there  a  better  approach  to  risk  management,  when  there  is  potential 
harm  to  the  government? 
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